Image deformation apparatus and method of controlling operation of same

ABSTRACT

A target image is deformed in such a manner that a subject in a reference image and a subject in the target image will coincide. A reference image and a target image are each divided into regions that conform to amounts of optical distortion. A region which is in the target image and which is common to the reference image and to the target image is subdivided into regions S35 to S37, S38 to S40, S42 to S44 in each of which amounts of optical distortion of both the reference image and target image are obtained. By using the amounts of optical distortion of the reference image and amounts of optical distortion of the target image  11  obtained from the subdivided regions S35 to S37, S38 to S40, S42 to S44, the target image is deformed in such a manner that a subject in the common region will coincide with the reference image.

CROSS-REFERENCE TO RELATES APPLICATIONS

This application is a Continuation of PCT International Application No.PCT JP2013/065033 filed on May 30, 2013, which claims priority under 35U.S.C. §119(a) to Japanese Patent Application No. 2012-157126 filed Jul.13, 2012. Each of the above application is hereby expressly incorporatedby reference, in its entirety, into the present application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an image deformation apparatus and to a methodof controlling the operation of this apparatus.

2. Description of the Related Art

In cases where a target image is combined with (superimposed upon) areference image, there is a possibility that multiplexing or a declinein resolution will be brought about after combining the images if thealignment accuracy of the reference image and target image is low. Inorder to improve alignment accuracy, it is necessary to deal not onlywith a positional shift of the entire image but also with a positionalshift of each individual subject that does not correspond to the shiftof the entire image. Lens distortion is known as a cause of suchshifting of individual images. Since lens distortion tends to becomemore pronounced the greater the distance from the optical center of thelens, the amount of distortion differs, even within the same image,depending upon position in the image. Further, if the angle of viewchanges, the position of a subject in the image changes as well and,hence, the amount of distortion of a subject shared by both thereference image and the target image will also differ.

FIG. 1 is an example of a reference image 1. The reference image 1includes the sun 2 as a subject at the upper left and a house 3 as asubject at the center. Since the house 3 is at the center, there islittle distortion ascribable to lens distortion. However, since the sun2 is at the upper left, there is a great amount of distortion caused bylens distortion.

FIG. 3 is an example of a target image 11. The target image 11 includesthe sun 12 as a subject that is slightly off-center and a house 13 as asubject at the lower right. Since the sun 12 is slightly off-center,there is little distortion ascribable to lens distortion. However, sincethe house 13 is at the lower right, there is a great amount ofdistortion caused by lens distortion.

There is a technique for suppressing a decline in resolution due to lensdistortion or a difference in lens fuselage at the time of imagecompositing, thereby producing a panorama image that is free of a senseof incongruity (Patent Document 1), a technique for detecting videodistortion, subjecting the distortion to a function approximation regionby region and compensating for distortion in each region using acalculated function (Patent Document 2), and a technique for generatinga blended motion-compensated image from a global motion-compensatedimage and a local motion-compensated image and combining a referenceimage with the blended motion-compensated image to thereby obtain a highnoise-reduction effect (Patent Document 3).

Furthermore, there is a technique for reducing amount of calculationwhile taking into consideration the distortion aberration of an opticalsystem (Patent Document 4) and a technique which, when an imaging deviceperforms split photography of a subject, sets the shooting directionappropriately in such a manner that the size of a superimposed regionwill be satisfactory at the time the individual captured images arejoined (Patent Document 5).

Patent Document 1: Japanese Patent Application Laid-Open No. 2010-161520

Patent Document 2: Japanese Patent Application Laid-Open No. 2011-49733

Patent Document 3: Japanese Patent Application Laid-Open No. 2011-147985

Patent Document 4: Japanese Patent Application Laid-Open No. 2007-13430

Patent Document 5: Japanese Patent Application Laid-Open No. 2011-139368

FIG. 33 is an example of an image obtained by combining the referenceimage 1 and the target image 11 in such a manner that the sun 2 andhouse 3 in the reference image 1 will coincide with the sun 12 and house13, respectively, of the target image 11. Since the amount of distortionof the sun 2 and the amount of distortion of the sun 12 differ as do theamount of distortion of the house 3 and the amount of distortion of thehouse 13, the sun images are misaligned and so are the house images.Thus, a composite image in which the subjects are not misaligned cannotbe obtained merely by combining the reference image 1 and the targetimage 11 in such a manner that identical subjects simply overlap eachother.

Patent Document 1 does not describe a concrete method for dealing withlocations where alignment is difficult owing to distortion caused bylens distortion. A composite image in which subjects are not misalignedcannot be obtained. Further, distortion is merely corrected for inPatent Document 2, and Patent Document 3 does not describe a regiondividing method for dealing with such causes of distortion as lensdistortion. A composite image in which the subjects are not misalignedcannot be obtained with the techniques taught in Patent Documents 4 or5.

SUMMARY OF THE INVENTION

An object of the present invention is to deform a target image so as toproduce a composite image in which subjects are not misaligned.

An image deformation apparatus according to the present inventioncomprises: an image input device (image input means) for inputtingmultiple frames of an image obtained by imaging the same subjectmultiple times (there are not only cases where the entire subject is thesame each time but also cases where a subject that is a portion of theentire subject is the same each time); a reference image decision device(reference image decision means) for deciding a reference image fromamong the multiple frames of the image that have been input from theimage input device; a target image decision device (target imagedecision means) for deciding a target image from among the multipleframes of the image other than the reference image that have been inputfrom the image input device; a region dividing device (region dividingmeans) for dividing the reference image decided by the reference imagedecision device and the target image decided by the target imagedecision device into regions that conform to amounts of opticaldistortion; a region subdividing device (region subdividing means) forsubdividing a common region, which is in the target image and is aregion common to the reference image and to the target image, intoregions in each of which both amount of optical distortion of thereference image and amount of optical distortion of the target image canbe partitioned, in accordance with the amounts of optical distortion inrespective ones of the regions of the reference image and regions of thetarget image divided by the region dividing device; and a deformationdevice (deformation means) for deforming the target image using theamounts of optical distortion of the reference image and amounts ofoptical distortion of the target image obtained from the regionssubdivided by the region subdividing device, and making a subject in thecommon region coincide with the reference image. (The target image isdeformed in such a manner that the target image will coincide with thereference image, but the target image need not necessarily coincideperfectly.)

The present invention also provides an operation control method suitedto the above-described image deformation apparatus. Specifically, thepresent invention provides a method of controlling operation of an imagedeformation apparatus, the method comprising steps of: an image inputdevice inputting multiple frames of an image obtained by imaging thesame subject multiple times; a reference image decision device decidinga reference image from among the multiple frames of the image that havebeen input from the image input device; a target image decision devicedeciding a target image from among the multiple frames of the imageother than the reference image that have been input from the image inputdevice; a region dividing device dividing the reference image decided bythe reference image decision device and the target image decided by thetarget image decision device into regions that conform to amounts ofoptical distortion; a region subdividing device subdividing a commonregion, which is in the target image and is a region common to thereference image and to the target image, into regions in each of whichboth amount of optical distortion of the reference image and amount ofoptical distortion of the target image can be partitioned, in accordancewith the amounts of optical distortion in respective ones of the regionsof the reference image and of the regions of the target image divided bythe region dividing device; and a deformation device deforming thetarget image using the amounts of optical distortion of the referenceimage and amounts of optical distortion of the target image obtainedfrom the regions subdivided by the region subdividing device, and makinga subject in the common region coincide with the reference image.

In accordance with the present invention, a region in the referenceimage and common to the reference image and to a target image issubdivided into regions in each of which both the amount of distortionof the reference image and the amount of distortion of the target imageare obtained. The target image is deformed so as to make a subject inthe common region coincide with the reference image by using the amountsof optical distortion of the reference image and of the target imageobtained from the regions subdivided. The image within the common regionin the deformed target image is deformed taking into consideration boththe optical distortion of the target image and the optical distortion ofthe reference image. As a result, when the target image is combined withthe reference image, the two images will coincide without misalignmentbetween a subject in the target image and a subject in the referenceimage.

The apparatus may further comprise a correction device (correctionmeans) for performing an optical-distortion correction in a case wherethe optical axis of an imaging optical system utilized in capturing thereference image and the target image is offset from centers of thereference image and target image, the optical-distortion correctionbeing performed centered on offset position of the optical axis. In thiscase, by way of example, the region dividing device divides thereference image and the target image, the amounts of optical distortionof which have been corrected by the correction device, into regions inaccordance with the corrected amounts of optical distortion.

The apparatus may further comprise an aligning device (aligning means)for aligning the reference image and the target image based upon amotion vector of the target image with respect to the reference image.In this case, by way of example, if the reference image and target imagehave been aligned by the aligning device, the region dividing devicewould, with regard to non-coincident portions of the images, divide thereference image and the target image into regions in accordance with thecorrected amounts of optical distortion.

The apparatus may further comprise an aligning device (aligning means)for aligning the reference image and the target image based upon amotion vector of the target image with respect to the reference image;and a determination device (determination means) for determining thatthe reference image or the target image contains a moving body if degreeof non-coincidence is equal to or greater than a predetermined value.

By way of example, the region dividing device divides the referenceimage and the target image into rectangular regions or concentriccircular regions that conform to amounts of optical distortion.

The apparatus may further comprise a compositing device (compositingmeans) for combining the reference image and the target image that havebeen deformed by the deformation device.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a reference image;

FIG. 2 is an example of the reference image divided into regions;

FIG. 3 is an example of a target image;

FIG. 4 is an example of the target image divided into regions;

FIG. 5 is an example of the target image subdivided into regions;

FIG. 6 illustrates the relationship between regions, which are obtainedby subdivision, and optical distortion;

FIG. 7 is an example of a deformed target image;

FIG. 8 illustrates the manner in which the reference image and thetarget image have been aligned;

FIG. 9 is a block diagram illustrating the electrical configuration ofan image sensing apparatus;

FIGS. 10 and 11 are flowcharts illustrating a portion of processingexecuted by the image sensing apparatus;

FIG. 12 is an example of a distortion intensity map;

FIGS. 13 and 14 are examples of the reference image and target image,respectively, divided into regions;

FIG. 15 is an example of the target image subdivided into regions;

FIG. 16 illustrates the relationship between regions, which are obtainedby subdivision, and optical distortion;

FIGS. 17 and 18 are flowcharts illustrating processing executed by theimage sensing apparatus;

FIG. 19 is an example of a distortion intensity map;

FIGS. 20 and 21 are flowcharts illustrating processing executed by theimage sensing apparatus;

FIG. 22 is an example of a deformed target image;

FIG. 23 illustrates the manner in which the reference image and thetarget image have been aligned;

FIG. 24 is an example of the target image;

FIG. 25 is an example of a deformed target image;

FIG. 26 is flowchart illustrating processing executed by the imagesensing apparatus;

FIG. 27 is an example of an allowable difference map;

FIG. 28 is an example of a reference image;

FIG. 29 is an example of a target image;

FIG. 30 is an example of a difference map;

FIG. 31 is an external view of a smart phone;

FIG. 32 is a block diagram illustrating the electrical configuration ofthe smart phone; and

FIG. 33 is an image obtained by superimposing a reference image and atarget image.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

According to this embodiment, a reference image and a target image aredecided from among subject images obtained by imaging performed multipletimes, and the target image is combined with (superimposed upon) thereference image. The reference image and target image are combined insuch a manner that subjects that are common to both the reference imageand the target image will coincide. In a case where the optical axis ofthe imaging lens and the center of the image of a subject obtained byimaging coincide, the amount of distortion increases from the center ofthe subject image toward the outer side thereof owing to opticaldistortion such as distortion ascribable to the imaging lens. In suchcase, even if the reference image and target image are combined,subjects included in common in both the reference image and the targetimage will not coincide owing to a difference between the amounts ofdistortion. In this embodiment, when a reference image and a targetimage are combined, subjects common to both the reference image and thetarget image are made to coincide.

FIG. 1 is an example of a reference image 1. The reference image 1includes the sun 2 as a subject at the upper left and a house 3 as asubject substantially at the center.

FIG. 2 illustrates the manner in which the reference image 1 is dividedinto regions.

As mentioned above, amount of distortion differs depending upon theregion of the reference image 1 owing to the distortion of the imaginglens. Generally, as set forth above, the closer to the center of theimage of the subject, the smaller the amount of optical distortion, withthe amount of optical distortion increasing as distance from the centerincreases.

The reference image 1 has been divided into nine regions (rather thannine regions, however, the number of regions may be smaller or largerthan nine) S11 to S19.

Regions S11, S13, S17 and S19 at the corners of the reference image 1are spaced away from the center and therefore exhibit a strong (large)amount of optical distortion, whereas the region S15 at the center ofthe reference image 1 exhibits a weak (small) amount of opticaldistortion. Regions S12, S14, S16 and S18 above, below and to the leftand right of the central region S15 exhibit a medium amount of opticaldistortion (an amount intermediate the amount of optical distortion ofthe central region S15 and amount of optical distortion of the fourcorner regions S11, S13, S17 and S19). These amounts of opticaldistortion are written within the parentheses near the region numbers.For example, the sun 2 exhibits a strong amount of optical distortionsince it is located in region S11 at the upper-left corner, and thehouse 3 exhibits a weak amount of optical distortion since it is locatedin the central region S15.

FIG. 3 is an example of the target image 11.

The target image 11 contains the sun 12 and house 13 as subjects. Thereare instances where the reference image 1 and target image 11 havedifferent angles of view owing to camera shake and the like. As aconsequence, even though the subjects the reference image 1 are the sameas those in the target image 11, there are instances where the relativepositions of the subjects in the subject images differ. For example,whereas the sun 2 is at the upper left in the reference image 1, the sun12 is somewhat closer to the center in the target image 11. Further,while the house 3 is substantially at the center in the reference image1, the house 13 is at the lower right in the target image 11.

FIG. 4 illustrates the manner in which the target image 11 is dividedinto regions.

In a manner similar to the reference image 1, the target image 11 hasbeen divided into nine regions S21 to S29. Naturally, the number ofregions of target image 11 may be smaller or larger than nine and thetarget image 11 may be divided into regions in a manner different fromthe reference image 1.

Regions S21, S23, S27 and S29 at the corners of the target image 11 arespaced away from the center and therefore exhibit a strong (large)amount of optical distortion, whereas the region S25 at the center ofthe target image 11 exhibits a weak (small) amount of opticaldistortion. Regions S22, S24, S26 and S28 above, below and to the leftand right of the central region S25 exhibit a medium amount of opticaldistortion (an amount intermediate the amount of optical distortion ofthe central region S25 and amount of optical distortion of the fourcorner regions S21, S23, S27 and 529). These amounts of opticaldistortion are written within the parentheses near the region numbers.For example, the sun 12 exhibits a weak amount of optical distortionsince it is located in central region S25. Since the house 13 is locatedin the regions S25, S26, S28 and S29, the amount of optical distortiondiffers depending upon the particular portion of the house 13.

Thus, even though the same subjects appear in both images, as in themanner of the sun 2 in the reference image 1 and the sun 12 in thetarget image 11 as well as the house 3 in the reference image 1 and thehouse 13 in the target image 11, the fact that the amounts of distortiondiffer means that even if the reference image 1 and the target image 11are combined so as to make identical images coincide, an offset will beproduced. That is, even if the sun 2 and the sun 12 are combined as wellas the house 3 and the house 13, they will not match and an offset willarise between them. In this embodiment, divided regions of the targetimage 11 are each subdivided in accordance with both the amount ofdistortion of the reference image 1 and amount of distortion of thetarget image 11. Since both the amount of distortion of the referenceimage 1 and amount of distortion of the target image 11 can beascertained in each of the subdivided regions, the target image 11, in acase where it is combined with the reference image 1, can be deformed soas to match the reference image 1 by using both the amount of distortionof the reference image 1 and amount of distortion of the target image11.

FIG. 5 is an example of the target image 11 after regions have beensubdivided.

When the reference image 1 divided into regions shown in FIG. 2 iscombined with the target image 11 divided into regions shown in FIG. 4in such a manner that identical subjects (the sun 2 and the sun 12, andthe house 3 and the house 13) are superimposed, the target image isfurther divided into finer regions (subdivided into these regions), asshown in FIG. 5, in each of which both amount of optical distortion ofthe reference image 1 and amount of optical distortion of the targetimage 11 can be classified, based upon the divided regions in referenceimage 1 shown in FIG. 2 and divided regions in target image 11 shown inFIG. 4.

By thus performing region subdivision in the target image 11, regionsS31 to S44 are produced in the target image 11.

FIG. 6 illustrates amounts of optical distortion in each region obtainedby region subdivision in the target image 11.

Although regions S31 to S44 are obtained by region subdivision asdescribed above, the amounts of distortion of the reference image 1 andthe amounts of distortion of the target image 11 are obtained only in acommon region 10, which is a region common to the reference image 1 andto the target image 11 representing the common portions of the images ofthe subjects. Only the common region 10 may be adopted as the target ofregion subdivision. In FIG. 6, therefore, amounts of distortion arewritten within the parentheses in each of the regions S35 to S37, S38 toS40 and S42 to S44 within the common region 10. The amounts ofdistortion are represented by the characters reading strong, medium andweak. STRONG indicates that the amount of distortion is strong, MEDIUMthat the amount of distortion is medium, and WEAK that the amount ofdistortion is weak. The first set of characters within the parenthesesindicates the amount of distortion of the reference image 1 in thatregion, and the second set of characters within the parenthesesindicates the amount of distortion of the target image 11 in thatregion. For example, since the amounts of distortion are written asbeing (STRONG, WEAK) in region S35, this indicates that the amount ofdistortion of reference image 1 and amount of distortion of target image11 corresponding to region S35 are strong and weak, respectively.Similarly, since the amounts of distortion are written as being (MEDIUM,WEAK) in region S36, this indicates that the amount of distortion ofreference image 1 and amount of distortion of target image 11corresponding to region S36 are medium and weak, respectively. Amountsof distortion are indicated similarly in the other regions.

FIG. 7 is an example of target image 11A after deformation thereof.

With regard to the regions within the common region 10, as shown in FIG.6, it is possible to ascertain not only the amounts of opticaldistortion of the target image 11 but also the amounts of opticaldistortion of the corresponding regions in the reference image 1.Therefore, by using both amounts of optical distortion, namely theamounts of optical, distortion of both the target image 11 and thereference image 1, the target image 11A is deformed in such a mannerthat the subjects (the sun 12 and the house 13) contained in the targetimage 11 will coincide with the subjects (the sun 2 and the house 3),which are contained in the reference image 1, in a case where the targetimage 11 has been combined with the reference image 1. The target image11 thus deformed is the deformed target image 11A illustrated in FIG. 7.

The target image 11A thus deformed contains the sun 12A and the house13A that coincide with the sun 2 and the house 3, which are distortedowing to lens distortion, contained in the reference image 1. Bycombining the deformed target image 11A and the reference image 1, acomposite image that is free of misalignment is obtained.

FIG. 8 illustrates the manner in which the deformed target image 11A andthe reference image 1 are combined.

Since the target image 11A has been deformed as described above, acomposite image that includes an offset-free sun 22 and an offset-freehouse 23 is obtained when the target image 11A is combined with thereference image 1.

Thus, even though optical distortion differs between the reference image1 and target image 11 from portion to portion, a composite image (imageresulting from superimposing images) containing offset-free subjects isobtained.

FIG. 9 is a block diagram illustrating the electrical configuration ofan image sensing apparatus.

The overall operation of the image sensing apparatus is controlled by aCPU 30.

The image sensing apparatus includes an operating unit 54 havingswitches, buttons and the like for inputting commands such as a shootingcommand and continuous shooting command. An operating signal that isoutput from the operating unit 54 is input to the CPU 30. The imagesensing apparatus further includes a light-emitting unit 35 forilluminating a subject and a light-receiving unit 36 for receiving lightreflected from the subject.

An imaging lens 31 is positioned in front of an image sensor 34. (Thereis a mechanical shutter in front of the imaging lens 31 but this is notillustrated.) An iris 32 and an optical low-pass filter 33 are disposedbetween the photoreceptor surface of the image sensor 34 and the imaginglens 31. The imaging lens 31 is positioned along the direction of theoptical axis by a lens driving unit 37, and the amount by which the iris32 is opened is controlled by an iris driving unit 38. A video signaloutput, etc., of the image sensor 34 is controlled by an image-sensordriving unit 39. The image sensing apparatus further includes acamera-shake detection unit 40 for detecting camera shake sustained bythe image sensor 34 and an image-stabilization driving unit 41 forshifting the image sensor 34.

The video signal that has been output from the image sensor 34 issubjected to predetermined analog signal processing such as whitebalance in an analog signal processing unit 42, which outputs theprocessed signal. The video signal that is output from the analog signalprocessing unit 42 is converted to digital image data by ananalog/digital conversion circuit 43.

The digital image data is recorded temporarily in a main memory 44 by amemory control unit 45. The digital image data is read out of the mainmemory 44 and is subjected to predetermined digital signal processingsuch as a gamma correction in a digital signal processing unit 46. Thedigital image data is input to a distortion/shading correction unit 47,subjected to a distortion correction and to a shading correction with areference position serving as the reference. The image data that hasbeen subjected to correction such as the distortion correction isapplied to a display unit 52 under the control of a display control unit53, whereby the image of the subject, which has been subjected tocorrection such as an optical distortion correction, is displayed on thedisplay screen of the display unit 52.

The digital image data read out of the main memory 44 is input to anintegration unit 49 as well. The integration unit 49 integrates theluminance component and adjusts the aperture value of the iris 32 basedupon the integrated value obtained.

When a record command is applied from the operating unit 54, the imagedata that has been corrected for optical distortion and the like asdescribed above is applied to and recorded on a memory card 50 under thecontrol of an external-memory control unit 51.

The deformation of the target image 11 and the combining of the deformedtarget image 11A and reference image 1 in the manner described above arecarried out in a case where continuous shooting has been performed. Itgoes without saying that these operations are not limited to instanceswhere continuous shooting has been performed and may be carried out inother cases as well.

FIGS. 10 and 11 are flowcharts illustrating processing executed by theimage sensing apparatus.

If processing for continuous-shooting compositing has not been set bythe operating unit 54 (“NO” at step 61), then single-shot shooting isperformed and the above-described compositing processing is notexecuted. If processing for continuous-shooting compositing has been set(“YES” at step 61), continuous shooting is carried out (step 62) andimage data representing multiple image frames is recorded temporarily inthe main memory 44. The image data representing multiple image framesobtained by continuous shooting is read out of the main memory 44 andinput to the distortion/shading correction unit 47, where the data issubjected to a distortion (optical distortion) correction and the like(step 63). The image data representing the multiple image frames thathave been subjected to a correction for distortion and the like areapplied to the display unit 52, and the multiple image framesrepresented by this image data are displayed on the display screen ofthe display unit 52. The operating unit 54 is utilized to decide areference image from among the multiple image frames displayed (step64). Further, a target image also is selected from among the multipleimage frames displayed (step 65).

As mentioned above, there are instances where even if a distortioncorrection is performed, distortion cannot be eliminated completely andsome will remain. Therefore, as described above (see FIGS. 2 and 4), thedecided reference image and the selected target image are each dividedinto regions in accordance with the intensity of distortion (step 66).Next, the reference image divided into regions and the target imagedivided into regions are aligned (combined, superimposed) in such amanner that subjects in regions of little distortion will coincide (step67).

If alignment succeeds in a region of weak distortion (“YES” at step 68),the region of the target image is subdivided based upon the referenceimage and target image that have been divided into regions in accordancewith intensity of distortion (step 69; see FIGS. 5 and 6), as describedabove. The target image is deformed using the intensity of distortion ofthe reference image and intensity of distortion of the target image inthe subdivided regions of the target image (step 70; FIG. 7).

If alignment does not succeed in a region of weak distortion (“NO” atstep 68), alignment is performed in a region of strong distortion (step72). If alignment succeeds in a region of strong distortion (“YES” atstep 73), then the target image is subdivided as described above (step69) and the target image is deformed (step 70). If alignment does notsucceed even in a region of strong distortion (“NO” at step 73), it isconstrued that the reference image cannot be combined with this targetimage and, hence, a different target image is selected (step 65).

The processing of steps 65 to 73 is repeated until processing ends withregard to all target images that are to be processed (step 71).

FIGS. 12 to 18 illustrate another embodiment. In this embodiment, adistortion intensity map is generated beforehand from informationconcerning the imaging lens 31 and the above-described division intoregions is performed utilizing the intensity map.

FIG. 12 is an example of a distortion intensity map 80.

The distortion intensity map 80 corresponds to the image of a subjectobtained by image capture and indicates the distortion intensities ofportions within the image of the subject. The distortion intensity map80 shown in FIG. 12 has been divided into three regions 75, 76 and 77 inthe form of concentric circles. The distortion in elliptical region 75at the center is weak, the distortion in the region 76 surrounding theregion 75 is medium, and the distortion in the outermost region 77 isstrong.

FIG. 13 is an example of reference image 1 divided into regions inaccordance with the distortion intensity map 80.

Boundaries 81 and 82 have been defined in the reference image 1 inaccordance with the distortion intensity map 80. Distortion in regionS31 inside boundary 81 is weak, distortion in region S32 located betweenboundaries 81 and 82 is medium, and distortion in region S33 outsideboundary 82 is strong.

FIG. 14 is an example of target image 11 divided into regions inaccordance with the distortion intensity map 80.

Boundaries 91 and 92 have been defined in the target image 11 as well inaccordance with the distortion intensity map 80 in a manner similar tothe reference image 1. Distortion in region S41 inside boundary 91 isweak, distortion in region S42 located between boundaries 91 and 92 ismedium, and distortion in region S43 outside boundary 92 is strong.

FIG. 15 is an example of the target image 11 after regions have beensubdivided.

When the reference image 1 divided into regions (see FIG. 13) and thetarget image 11 divided into regions (see FIG. 14) are obtained, asdescribed above, regions are subdivided based upon the reference image 1and the target image 11. As a result, in the subdivided regions, it ispossible to ascertain both the distortion intensity of every portion ofthe reference image 1 and distortion intensity of every portion of thetarget image 11.

FIG. 16 illustrates subdivided regions and distortion intensities in thetarget image 11.

As described above, regions S51 to S60 are defined in common region 10by subdividing the target image 11. In a manner similar to that shown inFIG. 6, distortion intensities are written within the parentheses ineach of the regions S51 to S60. The distortion intensities arerepresented by STRONG, MEDIUM and WEAK within the parentheses. Thedistortion intensity represented by the first set of characters withinthe parentheses indicates the distortion intensity in the region of thereference image 1 that corresponds to this region. The distortionintensity represented by the second set of characters within theparentheses indicates the distortion intensity of the target image.

FIGS. 17 and 18 are flowcharts illustrating processing executed by theimage sensing apparatus. Processing steps shown in FIGS. 17 and 18identical with those shown in FIG. 10 or 11 are designated by like stepnumbers and need not be described again.

As mentioned above, first a distortion intensity map (see FIG. 12) iscreated (step 101). A reference image is divided into regions inaccordance with the distortion intensity map created (step 102; see FIG.13). When a target image is selected (step 65), the selected targetimage is also divided into regions in accordance with the distortionintensity map (step 103). Processing following step 103 is similar tothat shown in FIG. 11.

FIGS. 19 and 20 illustrate another embodiment.

If the center of the reference image 1 or the target image 11 coincideswith the optical axis of the imaging lens 31, the greater the distancefrom the center, the more distortion intensity increases, as shown inFIG. 12. However, if the image sensor 34 is shifted as by imagestabilization, the center of the reference image 1 or the target image11 will no longer coincide with the optical axis of the imaging lens 31.Therefore, even if a distortion correction is applied in such a mannerthat, the greater the distance from the center of the reference image 1or the target image 11, the greater the strength of the correction (thisapplies to all subject images obtained by image capture, not just to thereference image 1 and target image 11), there will be instances where anappropriate distortion correction cannot be performed. If the imagesensor 34 has been shifted, this embodiment calculates a center positionof a distortion correction in accordance with the amount of shift andthe position to which the sensor has been shifted, and the distortioncorrection is applied using the calculated position as a reference.

FIG. 19 illustrates an example of distortion intensity map 80.

Regions 75, 76 and 77 have been defined such that, the greater thedistance from the center, the more distortion intensity increases, asshown in FIG. 12.

If center C1 of the image sensor 34 and the optical axis of the imaginglens 31 coincide, as indicated by dashed line 111, the image of asubject obtained by image capture by the image sensor 34 sustainsgreater distortion intensity as position in the image moves outward fromthe center. However, if the image sensor 34 is shifted so that center C2of the image sensor 34 is displaced from the optical axis of the imaginglens 31, as indicated by chain line 112, then the distortion intensityof the image of a subject obtained by image capture by the image sensor34 does not increase with distance from the center C2 of the image ofthe subject (which is identical with the center C2 of the image sensor34) but rather increases with distance from the optical axis C1 of theimaging lens 31. Distortion correction of the image of the subject mustbe carried out not by using the positional relationship with the centerC2 of the image sensor 34 as the reference but by using the positionalrelationship with the optical axis of the imaging lens 31 as thereference.

FIG. 20 is a flowchart illustrating distortion correction processing.

If continuous shooting is carried out (step 121), as mentioned above, animage to be subjected to distortion correction processing is selected(step 122).

Next, the position to which the image sensor 34 is shifted at the timeof shooting is calculated (step 123). The position to which the imagesensor 34 is shifted can be calculated based upon the amount of movementand direction of movement of the image sensor 34 in theimage-stabilization driving unit 41. Naturally, an arrangement may beadopted in which the image sensing apparatus is provided with a gyrosensor for detecting the position of the image sensor 34.

When the position to which the image sensor 34 has been shifted iscalculated, reference is had to the calculated shift position and to theposition of the optical axis of the imaging lens 31 to update the centerposition of the distortion correction with the position of the opticalaxis of the imaging lens 31 in the captured image serving as the centerposition of the distortion correction (step 124). The distortioncorrection of the selected image is carried out using the updated centerposition as the reference (step 125). In a case where the distortioncorrection is performed with regard to another image (“YES” at step126), the processing from steps 122 to 125 is repeated.

In the distortion correction in the foregoing embodiment, the sensorposition of the image sensor 34 is detected and the center position ofthe distortion correction is decided utilizing the detected sensorposition. However, it may be arranged so that the shift position of theimage sensor 34 is taken into consideration also in a case where theabove-described distortion intensity map is created. In such case,according to the distortion intensity map shown in FIG. 12, the greaterthe distance from the center of the optical axis, the greater thedistortion intensity will be.

FIGS. 21 to 25 illustrate another embodiment. In this embodiment,deformation is carried out in accordance with motion of the entiretarget image, the deformed target image and the reference image arecompared and a region where there is a difference is found. The regionwhere there is a difference is subjected to deformation processing usingthe distortion intensity of the reference image and the distortionintensity of the target image in the manner set forth above.

FIG. 21 is a flowchart illustrating processing executed by the imagesensing apparatus.

As described above, continuous shooting is performed (step 130) and adistortion correction is applied to multiple image frames thus obtained(step 131). Next, a reference image and a target image that is toundergo the above-described deformation processing are selected (step132). The selected target image is deformed upon estimating the motionof the entire target image (step 133), after which the reference imageand the target image are aligned and combined (step 134).

FIG. 22 is an example of a target image 11B deformed upon estimatingoverall motion thereof.

The target image 11B includes sun 12B and house 13B.

FIG. 23 illustrates the reference image 1 and the target image 11B afterthey have been aligned.

Since the target image 11B has been deformed upon estimating its overallmotion, the house 3 contained in the reference image 1 and the house 13Bcontained in the target image 11B coincide when the reference image 1and target image 11B are aligned. However, an offset develops betweenthe sun 2 contained in the reference image 1 and the sun 12B containedin the target image 11B because the distortion intensity of the sun 2and that of the sun 12B differ.

With reference again to FIG. 21, the difference is obtained between thereference image 1 and the target image 11B (step 135) and the targetimage is divided into a region in which there is a difference (offset)and a region in which there is no difference (step 136).

With reference to FIG. 23, by obtaining the difference between thereference image 1 and the target image 11B, the region 10 common to thereference image 1 and to the target image 11B is divided into a regionS61 in which there is a difference and a region S62 in which there is nodifference. Thus it may be arranged so that the reference image 1 andtarget image 11B are aligned based upon a motion vector of the targetimage 11B with respect to the reference image 1. (This is carried out bythe aligning means.)

FIG. 24 is an example of the target image 11 prior to deformation takingoverall offset into consideration.

When the region 561 in which there is a difference is detected byaligning the reference image 1 and the deformed target image 11B, asshown in FIG. 23, the region 10 common to the reference image 1 and tothe target image 11 is divided into a region S71, which corresponds tothis region 561 in which there is a difference, and a region S72, whichcorresponds to the region S62 in which there is no difference.

With reference again to FIG. 21, when the target image 11 prior todeformation is divided into the region S71 corresponding to the regionS61 in which there is a difference and the region S72 corresponding tothe region S62 in which there is no difference (step 137), the imagewithin the region S71 corresponding to region S61 in which there is adifference is deformed so as to coincide with the reference image 1 byusing the distortion intensity of the reference image 1 and thedistortion intensity of the target image 11 (step 138).

FIG. 25 illustrates an example target image 11B after deformationthereof.

As shown in FIG. 23, a region 381 in which there is a difference betweenthe target image 113 and the reference image 1 and a region 382 in whichthere is no difference are defined by aligning the images.

The region S72 corresponding to the region in which there is nodifference in the target image 11 prior to deformation as shown in FIG.24 is replaced by the region S82 corresponding to the region in whichthere is no difference in the target image 11B deformed as shown in FIG.25. With regard to the region S71 shown in FIG. 24, it is deformed so asto coincide with the reference image 1 by using the distortion intensityof the reference image 1 and the distortion intensity of the targetimage 11 (step 139 in FIG. 21) in a manner similar to that describedabove.

In a case where similar processing is executed with regard to otherimages (“YES” at step 140), processing from step 132 onward is repeated.

FIGS. 26 to 30 illustrate a further embodiment. This embodiment detectswhether a reference image or a target image contains a moving body.

FIG. 26 is a flowchart illustrating processing executed by the imagesensing apparatus.

First, an allowable difference map is generated (step 170).

FIG. 27 is an example of an allowable difference map 160.

As described above, the difference is obtained between a reference imageand a target image the entirety of which has been deformed takingoverall misalignment into account. When this difference has beenobtained, the allowable difference map 160 indicates how much differenceis allowable.

A plurality of concentric, circular regions 161, 162 and 163 have beendefined in the difference allowance map 160. The ring-shaped region 162has been defined so as to surround the outside of the central region161, and the region 163 has been defined around the region 162. Theregions 161, 162 and 163 indicate allowable amounts of difference; thesmaller the allowable amount of difference, the finer the hatching. Theallowable amount of difference is the smallest for region 161, the nextsmallest for region 162 and the greatest for region 163. This is to takeinto consideration the fact that, since distortion intensity becomesprogressively larger from the center to the outer side of an image, thedifference also increases.

With reference again to FIG. 26, a reference image and a target imageare selected (step 171).

FIG. 28 is an example of a reference image 1A.

The reference image 1A contains sun 2A, house 3A and automobile 4A.

Next, in a manner similar to that described above, the entire referenceimage is deformed (step 172 in FIG. 26). The target image is alignedwith the reference image in conformity with the overall offset (step173).

FIG. 29 is an example of a target image 11C.

The target image 11C shown in FIG. 29 contains sun 12C, house 13C andautomobile 141C. The target image 11C has been deformed (shifted) in itsentirety in such a manner that the sun 12C coincides with the sun 2A ofthe reference image 1A shown in FIG. 28 and the house 13C coincides withthe house 13A shown in FIG. 28. Thus, since corresponding subjects areat matching positions, the reference image 1A and the target image 11Ccan be aligned as set forth above.

The difference between the reference image 1A and the target image 11Cis obtained by aligning the images (step 174).

FIG. 30 is an example of a difference map 150 indicating differentialvalues.

The smaller the difference in the difference map 150, the finer thehatching.

The difference map 150 includes regions 151, 152, 153, 154 and 155.Region 151 indicates the difference between the background of thereference image 1A and the background of the target image 11C, region152 indicates the difference between the sun 2A of the reference image1A and the sun 12C of the target image 11C, region 153 indicates thedifference between the house 3A of the reference image 1A and the house13C of the target image 11C, region 154 indicates the difference betweenthe automobile 4A of the reference image 1A and the background of thetarget image 11C, and region 155 indicates the difference between thebackground of the reference image 1A and the automobile 14C of thetarget image 11C. Since the automobile is moving between the moment ofcapture of the reference image 1A and the moment of capture of thetarget image 11C, there is a considerable difference between theposition of the automobile 4A in the reference image 1A and theautomobile 14C in the target image 11C. Consequently, the differencebetween the region 154 and the region 155 is large.

When the differences are obtained as shown in FIG. 30, a comparison ismade with the allowable difference map 160, which is shown in FIG. 27,for every difference region 151, 152, 153, 154 and 155 (step 173).

If the result of the comparison between the differences and theallowable difference map 160 is that a difference is less than atolerance value in any region (“YES” at step 176), then a flag to theeffect that a moving body does not exist is added to the target image(step 177). If there is a location where the difference is equal to orgreater than the tolerance value (“NO” at step 176), then a flag to theeffect that a moving body exists is added to the target image (step178). Thus the reference image 1 and the target image 11C are alignedbased upon a motion vector of the target image 11C with respect to thereference image 1A and, if the extent of non-coincidence is equal to orgreater than a predetermined amount, it is judged that the referenceimage 1 or the target image 11C contains a moving body.

In a case where processing regarding another target image is repeated,the processing from step 131 onward is executed (step 179).

A case where this embodiment has been applied to an image sensingapparatus is described above. However, the invention can also beapplied, for example, to a built-in or externally mounted camera for apersonal computer, or to a mobile terminal device, such as thatdescribed below, having a camera function.

A mobile telephone, a smart phone, a PDA (Personal Digital Assistant)and a mobile game device can be mentioned as examples of a mobileterminal device this embodiment. The invention will now be described indetail with reference to the drawings taking a smart phone as anexample.

FIG. 31 illustrates the external appearance of a smart phone 181according to an embodiment of an image sensing apparatus in accordancewith the present invention.

The smart phone 181 shown in FIG. 31 has a case 182 in the shape of aflat plate, and one side face of the case 182 is equipped with a displayinput unit 190 provided, in the form of a unitary body, with a displaypanel 191 serving as a display section and with an operating panel 192serving as an input section. The case 182 is further equipped with amicrophone 202, a speaker 201, an operating device 220 and a camera unit221. The configuration of the case 182 is not limited to that shownhere. For example, the case can have a configuration in which a displayunit and an input unit are separate from each other, or a configurationhaving a clam-shell structure or a sliding mechanism.

FIG. 32 is a block diagram illustrating the configuration of the smartphone 181 shown in FIG. 31.

As shown in FIG. 32, the main components of the smart phone are awireless communication unit 205, the display input unit 190, aconversation unit 200, the operating device 220, the camera unit 221, astorage unit 230, an external input/output unit 24C, a GPS (GlobalPositioning System) receiving unit 250, a motion sensor 260, a powersupply unit 270 and a main control unit 280. Further, the smart phone181 is equipped with a wireless communication function, which is themain function of the smart phone 181, for performing mobile wirelesscommunication via a base station unit BS and a mobile communicationnetwork NW.

In accordance with a command from the main control unit 280, thewireless communication unit 205 performs wireless communication with thebase station BS accommodated in the mobile communication network NW.Using such wireless communication, the wireless communication unit 205sends and receives various file data such as voice data and image dataas well as email data, and receives data such as World-Wide Web data andstreaming data.

The display input unit 190 is equipped with the display panel 191 andoperating panel 142 in the form of a so-called “touch panel” which,under the control of the main control unit 280, displays images (stillimages and moving images) and text information and the like to therebyconvey information to the user visually, and detects operations made bythe user in response to the information displayed.

The display panel 191 employs a display cell such as an LCD (LiquidCrystal Display) or an OELD (Organic Electro-Luminescence Display) as adisplay device. The operating panel 192 is a device on which an image,which is displayed on the display screen of the display panel 191, isviewably placed, and detects one or multiple coordinates. It is operatedby the user's finger or by a stylus. When this device is touched by theuser's finger or by a stylus, the device outputs a detection signal,which is generated due to such operation, to the main control unit 280.Next, based upon the detection signal received, the main control unit280 detects the position (coordinates) touched on the display panel 191.

As illustrated in FIG. 31, the display panel 191 and the operating panel192 of the smart phone 181 exemplified as an embodiment of an imagesensing apparatus are integrated into a whole to thereby construct thedisplay input unit 190. The operating panel 192 is placed so as tocompletely cover the display panel 191. In a case where such asarrangement is adopted, the operating panel 192 may be equipped with afunction for detecting user operator even with regard to an areaexterior to the display panel 191. In other words, the operating panel192 may just as well be equipped with a detection area (referred to as a“display area” below) with regard to the portion that overlaps thedisplay panel 191, and with a detection area (referred to as a“non-display area” below) with regard to the fringe portion that doesnot overlap the display panel 191.

It should be noted that although the size of the display area and thesize of the display panel 191 may coincide perfectly, the two need notnecessarily coincide. Further, the operating panel 192 may be equippedwith two sensitive regions, namely one on the fringe portion and one onthe portion inwardly thereof. Furthermore, the width of the fringeportion is designed appropriately in accordance with the size of thecase 182. In addition, systems such as a matrix switch system,resistive-film system, surface elastic wave system, infrared system,electromagnetic induction system and electrostatic capacitance systemcan be mentioned as position detection systems employed by the operatingpanel 192, and any of these systems can be adopted.

The conversation unit 200, which has the speaker 201 and the microphone202, converts the user's voice, which has entered through the microphone202, to voice data processable by the main control unit 280, decodesvoice data, which has been received by the wireless communication unit205 or external input/output unit 240, and outputs the decoded data fromthe speaker 201. Further, as shown in FIG. 31, and by way of example,the speaker 201 can be mounted on the same face as the face providedwith the display input unit 190, and the microphone 202 be mounted onthe side face of the case 182.

The operating device 220, which is a hardware key using a key switch orthe like, accepts commands from the user. For example, as shown in FIG.31, the operating device 220 is a push-button switch mounted on the sideface of the case 182 of the smart phone 181. The switch is turned ON bybeing pressed by a finger or the like and is restored to the OFF stateby the restoration force of a spring or the like when the finger isremoved.

The storage unit 230 stores the control program and control data of themain control unit 280, application software, address data associatedwith the names and telephone numbers, etc., of communicating parties,email data sent and received, Web data downloaded by Web browsing, anddownloaded content, and also stores streaming data temporarily. Further,the storage unit 230 is constituted by an internal storage device 231,which is built in the smart phone, and a removable external storagedevice 232 having an external memory slot. The internal storage device231 and external storage device 232 constructing the storage unit 230are implemented using storage media such as a flash memory, hard disk, amultimedia card micro-type memory or card-type memory [e.g., Micro SD(registered trademark) memory or the like], RAM (Random-Access Memory)and ROM (Read-Only Memory).

The external input/output unit 240, which functions as the interfacewith all external devices connected to the smart phone 181, is forconnecting directly or indirectly other external devices as bycommunication [such as Universal Serial Bus (USB) or IEEE 1394] ornetwork [e.g., Internet, wireless LAN (Local-Area Network), Bluetooth(registered trademark), REID (Radio-Frequency Identification), InfraredData Association: IrDA (registered trademark), UWB (Ultra-Wideband(registered trademark) or Zigbee (registered trademark)].

Examples of devices connected to the smart phone 181 are awired/wireless headset; wired/wireless external charging device;wired/wireless data port; memory card or SIM (Subscriber Identity ModuleCard)/UIM (User Identity Module) card connected via a card socket;external audio/video device connected via an audio/video I/O(Input/Output) terminal; wirelessly connected external audio/videodevice; wired/wireless connected smart phone; wired/wirelessly connectedpersonal computer; wired/wirelessly connected PDA; wired/wirelesslyconnected personal computer; and earphone. The external input/outputsection can be adapted so as to transmit data, which has been receivedfrom such external devices, to each component within the smart phone181, and so as to transmit data within the smart phone 181 to theexternal devices.

In accordance with a command from the main control unit 280, the GPSreceiving unit 250 receives GPS signals sent from GPS satellites ST1 toSTn, executes positioning processing that is based upon multiple GPSsignals received, and detects position comprising the longitude,latitude and elevation of the smart phone 181. When position informationis capable of being acquired from the wireless communication unit 205 orthe external input/output unit 240 (e.g., wireless LAN), the GPSreceiving unit 250 can also detect position using this positioninformation.

The motion sensor 260, which has a three-axis acceleration sensor, forexample, detects physical motion of the smart phone 181 in accordancewith a command from the main control unit 280. The traveling directionand acceleration of the smart phone 181 are detected by detecting thephysical motion of the smart phone 181. The result of such detection isoutput to the main control unit 280.

In accordance with a command from the main control unit 280, the powersupply unit 270 supplies each unit of the smart phone 181 with powerstored in a battery (not shown).

The main control unit 280, which is equipped with a microprocessor,operates in accordance with a control program and control data stored bythe storage unit 230 and controls overall operation of each unit of thesmart phone 181. Further, the main control unit 280 has a mobilecommunication control function, which controls each unit of thecommunication system, and an application processing function, in orderto perform voice communication and data communication through thewireless communication unit 205.

The application processing function is implemented by operation of themain control unit 280 in accordance with application software stored bythe storage unit 230. Examples of the application processing functionare an infrared communication function for communicating data with anopposing device by controlling the external input/output unit 240, anemail function for sending and receiving email, and a Web browsingfunction for viewing Web pages.

The main control unit 280 has an image processing function fordisplaying video on the display input unit 190 based upon received dataand image data (still-image data and moving-image data) such asdownloaded streaming data. The image processing function refers to afunction whereby the main control unit 280 decodes the above-mentionedimage data, applies image processing to the decoded result and displaysthe resultant image on the display input unit 190.

The main control unit 280 further executes display control of thedisplay panel 191 and operation detection control for detecting useroperation through the operating device 220 and operating panel 192.

By executing display control, the main control unit 280 displays iconsfor launching application software, software keys such as a scroll bar,or displays a window for creating email. It should be noted that thescroll bar refers to a software key for accepting a command, which movesa displayed portion of an image, with regard to an image too large tofit in the display area of the display panel 191.

Further, by executing operation detection control, the main control unit280 detects user operation performed via the operating device 220, oraccepts tapping of the icons and character-string inputs to an inputsection of the above-mentioned window through the operation panel 182,or accepts a displayed-image scroll request issued through the scrollbar.

Furthermore, the main control unit 280 has a touch-panel controlfunction which, through execution of the operation detection function,determines whether a position touched on the operating panel 192 is aportion (the display area) that overlaps the display panel 191 or afringe portion (the non-display area) that does not overlap the displaypanel 191, and controls the sensitive region of the operating panel 192and the display positions of software keys.

Further, the main control unit 280 detects gestures applied to theoperating panel 192 and is capable of executing preset functions inaccordance with a detected gesture. Here a gesture refers not to asimple, conventional touching operation but to the tracing of a path bya finger or the like, the designation of multiple positionssimultaneously, or an operation which, by combining these, traces a pathwith regard to at least one of multiple positions.

The camera unit 221 is a digital camera for performing electronicshooting using a CMOS (Complementary Metal-Oxide Semiconductor) or CCD(Charge-Coupled Device) or the like. Further, under control exercised bythe main control unit 280, the camera unit 221 converts image dataobtained by imaging to compressed image data such as JPEG (JointPhotographic coding Experts Group) data and is capable of storing thecompressed image data in the storage unit 230 or of outputting the datathrough the external input/output unit 240 or wireless communicationunit 205. In the smart phone 181 shown in FIG. 31, the camera unit 221has been mounted on the same side as that having the display input unit190. However, the mounting position of the camera unit 221 is notlimited to that shown. The camera unit 221 may be mounted on the backside of the display input unit 190, and it is permissible to mount aplurality of camera units 221. It should be noted that in a case where aplurality of the camera units 221 have been mounted, the camera units221 used in shooting may be switched among and used singly, or shootingmay be performed by using the plurality of camera its 221simultaneously.

Further, the camera unit 221 can be utilized for various functionspossessed by the smart phone 181. For example, an image acquired by thecamera unit 221 can be displayed on the display panel 191, and the imagefrom the camera unit 221 can be utilized as one operational input on theoperating panel 192. Further, when the GPS receiving unit 250 detectsposition, position can also be detected by referring to the image fromthe camera unit 221. Furthermore, by referring to the image from thecamera unit 221, the optical-axis direction of the camera unit 221 ofthe smart phone 181 can be determined without using a three-axisacceleration sensor or in conjunction with a three-axis accelerationsensor, and the present environment of use can be determined. Naturally,the image from the camera unit 221 can be utilized within theapplication software as well.

In addition, such information as position information acquired by theGPS receiving unit 250, voice information acquired by the microphone 202(which may be text information obtained by a voice-to-text conversionperformed by the main control unit or the like), and attitudeinformation acquired by the motion sensor 260 can be appended tostill-image or moving-mage data and the result can be stored in thestorage unit 230 or can be output through the external input/output unit240 or wireless communication unit 205.

In a case where the processing according to this embodiment is appliedto the smart phone 181, the above-described processing would be executedby the main control unit 280.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

What is claimed is:
 1. An image deformation apparatus comprising: animage input device for inputting multiple frames of an image obtained byimaging the same subject multiple times; a reference image decisiondevice for deciding a reference image from among the multiple frames ofthe image that have been input from said image input device; a targetimage decision device for deciding a target image from among themultiple frames of the image other than the reference image that havebeen input from said image input device; a region dividing device fordividing the reference image decided by said reference image decisiondevice and the target image decided by said target image decision deviceinto regions that conform to amounts of optical distortion; a regionsubdividing device for subdividing a common region, which is in thetarget image and is a region common to the reference image and to thetarget image, into regions in each of which both amount of opticaldistortion of the reference image and amount of optical distortion ofthe target image can be partitioned, in accordance with the amounts ofoptical distortion in respective ones of the regions of the referenceimage and regions of the target image divided by said region dividingdevice; and a deformation device for deforming the target image usingthe amounts of optical distortion of the reference image and amounts ofoptical distortion of the target image obtained from the regionssubdivided by said region subdividing device, and making a subject inthe common region coincide with the reference image.
 2. The apparatusaccording to claim 1, further comprising a correction device forperforming an optical-distortion correction in a case where the opticalaxis of an imaging optical system utilized in capturing the referenceimage and the target image is offset from centers of the reference imageand target image, the optical-distortion correction being performedcentered on offset position of the optical axis; wherein said regiondividing device divides the reference image and the target image, theamounts of optical distortion of which have been corrected by thecorrection device, into regions in accordance with the corrected amountsof optical distortion.
 3. The apparatus according to claim 1, furthercomprising an aligning device for aligning the reference image and thetarget image based upon a motion vector of the target image with respectto the reference image; wherein if the reference image and target imagehave been aligned by said aligning device, said region dividing devicedivides, with regard to non-coincident portions of the images, thereference image and the target image into regions in accordance with thecorrected amounts of optical distortion.
 4. The apparatus according toclaim 1, further comprising: an aligning device for aligning thereference image and the target image based upon a motion vector of thetarget image with respect to the reference image; and a determinationdevice for determining that the reference image or the target imagecontains a moving body if degree of non-coincidence is equal to orgreater than a predetermined value.
 5. The apparatus according to claim1, wherein said region dividing device divides the reference image andthe target image into rectangular regions or concentric circular regionsthat conform to amounts of optical distortion.
 6. The apparatusaccording to claim 1, further comprising a compositing device forcombining the reference image and the target image that have beendeformed by said deformation device.
 7. A method of controllingoperation of an image deformation apparatus, said method comprisingsteps of: an image input device inputting multiple frames of an imageobtained by imaging the same subject multiple times; a reference imagedecision device deciding a reference image from among the multipleframes of the image that have been input from the image input device; atarget image decision device deciding a target image from among themultiple frames of the image other than the reference image that havebeen input from the image input device; a region dividing devicedividing the reference image decided by the reference image decisiondevice and the target image decided by the target image decision deviceinto regions that conform to amounts of optical distortion; a regionsubdividing device subdividing a common region, which is in the targetimage and is a region common to the reference image and to the targetimage, into regions in each of which both amount of optical distortionof the reference image and amount of optical distortion of the targetimage can be partitioned, in accordance with the amounts of opticaldistortion in respective ones of the regions of the reference image andregions of the target image divided by the region dividing device; and adeformation device deforming the target image using the amounts ofoptical distortion of the reference image and amounts of opticaldistortion of the target image obtained from the regions subdivided bythe region subdividing device, and making a subject in the common regioncoincide with the reference image.